{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');\n",
       "@import url('http://fonts.googleapis.com/css?family=Vollkorn');\n",
       "@import url('http://fonts.googleapis.com/css?family=Arimo');\n",
       "@import url('http://fonts.googleapis.com/css?family=Fira_sans');\n",
       "\n",
       "    div.cell{\n",
       "        width: 900px;\n",
       "        margin-left: 0% !important;\n",
       "        margin-right: auto;\n",
       "    }\n",
       "    div.text_cell code {\n",
       "        background: transparent;\n",
       "        color: #000000;\n",
       "        font-weight: 600;\n",
       "        font-size: 11pt;\n",
       "        font-style: bold;\n",
       "        font-family:  'Source Code Pro', Consolas, monocco, monospace;\n",
       "   }\n",
       "    h1 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "\t}\n",
       "\t\n",
       "    div.input_area {\n",
       "        background: #F6F6F9;\n",
       "        border: 1px solid #586e75;\n",
       "    }\n",
       "\n",
       "    .text_cell_render h1 {\n",
       "        font-weight: 200;\n",
       "        font-size: 30pt;\n",
       "        line-height: 100%;\n",
       "        color:#c76c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 1em;\n",
       "        display: block;\n",
       "        white-space: wrap;\n",
       "        text-align: left;\n",
       "    } \n",
       "    h2 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "        text-align: left;\n",
       "    }\n",
       "    .text_cell_render h2 {\n",
       "        font-weight: 200;\n",
       "        font-size: 16pt;\n",
       "        font-style: italic;\n",
       "        line-height: 100%;\n",
       "        color:#c76c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 1.5em;\n",
       "        display: block;\n",
       "        white-space: wrap;\n",
       "        text-align: left;\n",
       "    } \n",
       "    h3 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "    }\n",
       "    .text_cell_render h3 {\n",
       "        font-weight: 200;\n",
       "        font-size: 14pt;\n",
       "        line-height: 100%;\n",
       "        color:#d77c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 2em;\n",
       "        display: block;\n",
       "        white-space: wrap;\n",
       "        text-align: left;\n",
       "    }\n",
       "    h4 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "    }\n",
       "    .text_cell_render h4 {\n",
       "        font-weight: 100;\n",
       "        font-size: 14pt;\n",
       "        color:#d77c0c;\n",
       "        margin-bottom: 0.5em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "        white-space: nowrap;\n",
       "    }\n",
       "    h5 {\n",
       "        font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "    }\n",
       "    .text_cell_render h5 {\n",
       "        font-weight: 200;\n",
       "        font-style: normal;\n",
       "        color: #1d3b84;\n",
       "        font-size: 16pt;\n",
       "        margin-bottom: 0em;\n",
       "        margin-top: 0.5em;\n",
       "        display: block;\n",
       "        white-space: nowrap;\n",
       "    }\n",
       "    div.text_cell_render{\n",
       "        font-family: 'Fira sans', verdana,arial,sans-serif;\n",
       "        line-height: 125%;\n",
       "        font-size: 115%;\n",
       "        text-align:justify;\n",
       "        text-justify:inter-word;\n",
       "    }\n",
       "    div.output_subarea.output_text.output_pyout {\n",
       "        overflow-x: auto;\n",
       "        overflow-y: scroll;\n",
       "        max-height: 50000px;\n",
       "    }\n",
       "    div.output_subarea.output_stream.output_stdout.output_text {\n",
       "        overflow-x: auto;\n",
       "        overflow-y: scroll;\n",
       "        max-height: 50000px;\n",
       "    }\n",
       "    div.output_wrapper{\n",
       "        margin-top:0.2em;\n",
       "        margin-bottom:0.2em;\n",
       "}\n",
       "\n",
       "    code{\n",
       "      font-size: 70%;\n",
       "    }\n",
       "    .rendered_html code{\n",
       "    background-color: transparent;\n",
       "    }\n",
       "    ul{\n",
       "        margin: 2em;\n",
       "    }\n",
       "    ul li{\n",
       "        padding-left: 0.5em; \n",
       "        margin-bottom: 0.5em; \n",
       "        margin-top: 0.5em; \n",
       "    }\n",
       "    ul li li{\n",
       "        padding-left: 0.2em; \n",
       "        margin-bottom: 0.2em; \n",
       "        margin-top: 0.2em; \n",
       "    }\n",
       "    ol{\n",
       "        margin: 2em;\n",
       "    }\n",
       "    ol li{\n",
       "        padding-left: 0.5em; \n",
       "        margin-bottom: 0.5em; \n",
       "        margin-top: 0.5em; \n",
       "    }\n",
       "    ul li{\n",
       "        padding-left: 0.5em; \n",
       "        margin-bottom: 0.5em; \n",
       "        margin-top: 0.2em; \n",
       "    }\n",
       "    a:link{\n",
       "       font-weight: bold;\n",
       "       color:#447adb;\n",
       "    }\n",
       "    a:visited{\n",
       "       font-weight: bold;\n",
       "       color: #1d3b84;\n",
       "    }\n",
       "    a:hover{\n",
       "       font-weight: bold;\n",
       "       color: #1d3b84;\n",
       "    }\n",
       "    a:focus{\n",
       "       font-weight: bold;\n",
       "       color:#447adb;\n",
       "    }\n",
       "    a:active{\n",
       "       font-weight: bold;\n",
       "       color:#447adb;\n",
       "    }\n",
       "    .rendered_html :link {\n",
       "       text-decoration: underline; \n",
       "    }\n",
       "    .rendered_html :hover {\n",
       "       text-decoration: none; \n",
       "    }\n",
       "    .rendered_html :visited {\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .rendered_html :focus {\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .rendered_html :active {\n",
       "      text-decoration: none;\n",
       "    }\n",
       "    .warning{\n",
       "        color: rgb( 240, 20, 20 )\n",
       "    } \n",
       "    hr {\n",
       "      color: #f3f3f3;\n",
       "      background-color: #f3f3f3;\n",
       "      height: 1px;\n",
       "    }\n",
       "    blockquote{\n",
       "      display:block;\n",
       "      background: #fcfcfc;\n",
       "      border-left: 5px solid #c76c0c;\n",
       "      font-family: 'Open sans',verdana,arial,sans-serif;\n",
       "      width:680px;\n",
       "      padding: 10px 10px 10px 10px;\n",
       "      text-align:justify;\n",
       "      text-justify:inter-word;\n",
       "      }\n",
       "      blockquote p {\n",
       "        margin-bottom: 0;\n",
       "        line-height: 125%;\n",
       "        font-size: 100%;\n",
       "      }\n",
       "</style>\n",
       "<script>\n",
       "    MathJax.Hub.Config({\n",
       "                        TeX: {\n",
       "                           extensions: [\"AMSmath.js\"]\n",
       "                           },\n",
       "                tex2jax: {\n",
       "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
       "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
       "                },\n",
       "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
       "                \"HTML-CSS\": {\n",
       "                    scale:100,\n",
       "                        availableFonts: [],\n",
       "                        preferredFont:null,\n",
       "                        webFont: \"TeX\",\n",
       "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
       "                }\n",
       "        });\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# format the book\n",
    "import sys\n",
    "sys.path.insert(0,'../code')\n",
    "import book_format\n",
    "book_format.load_style('../code')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bayes’s Theorem\n",
    "\n",
    "## Conditional probability\n",
    "\n",
    "The fundamental idea behind all Bayesian statistics is Bayes’s theorem,\n",
    "which is surprisingly easy to derive, provided that you understand\n",
    "conditional probability. So we’ll start with probability, then\n",
    "conditional probability, then Bayes’s theorem, and on to Bayesian\n",
    "statistics.\n",
    "\n",
    "A probability is a number between 0 and 1 (including both) that\n",
    "represents a degree of belief in a fact or prediction. The value 1\n",
    "represents certainty that a fact is true, or that a prediction will come\n",
    "true. The value 0 represents certainty that the fact is false.\n",
    "\n",
    "Intermediate values represent degrees of certainty. The value 0.5, often\n",
    "written as 50%, means that a predicted outcome is as likely to happen as\n",
    "not. For example, the probability that a tossed coin lands face up is\n",
    "very close to 50%.\n",
    "\n",
    "A conditional probability is a probability based on some background\n",
    "information. For example, I want to know the probability that I will\n",
    "have a heart attack in the next year. According to the CDC, “Every year\n",
    "about 785,000 Americans have a first coronary attack.\n",
    "(<http://www.cdc.gov/heartdisease/facts.htm>)”\n",
    "\n",
    "The U.S. population is about 311 million, so the probability that a\n",
    "randomly chosen American will have a heart attack in the next year is\n",
    "roughly 0.3%.\n",
    "\n",
    "But I am not a randomly chosen American. Epidemiologists have identified\n",
    "many factors that affect the risk of heart attacks; depending on those\n",
    "factors, my risk might be higher or lower than average.\n",
    "\n",
    "I am male, 45 years old, and I have borderline high cholesterol. Those\n",
    "factors increase my chances. However, I have low blood pressure and I\n",
    "don’t smoke, and those factors decrease my chances.\n",
    "\n",
    "Plugging everything into the online calculator at\n",
    "<http://cvdrisk.nhlbi.nih.gov/calculator.asp>, I find that my risk of a\n",
    "heart attack in the next year is about 0.2%, less than the national\n",
    "average. That value is a conditional probability, because it is based on\n",
    "a number of factors that make up my “condition.”\n",
    "\n",
    "The usual notation for conditional probability is\n",
    "<span>$\\mathrm{p}(A|B)$</span>, which is the probability of $A$ given\n",
    "that $B$ is true. In this example, $A$ represents the prediction that I\n",
    "will have a heart attack in the next year, and $B$ is the set of\n",
    "conditions I listed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conjoint probability\n",
    "\n",
    "<span>**Conjoint probability**</span> is a fancy way to say the\n",
    "probability that two things are true. I write\n",
    "<span>$\\mathrm{p}(A {~\\mathrm{and}~}B)$</span> to mean the probability\n",
    "that $A$ and $B$ are both true.\n",
    "\n",
    "If you learned about probability in the context of coin tosses and dice,\n",
    "you might have learned the following formula:\n",
    "\n",
    "$${{\\mathrm{p}(A {~\\mathrm{and}~}B)}} = {{\\mathrm{p}(A)}}~{{\\mathrm{p}(B)}} \\quad\\quad\\mbox{WARNING: not always true}$$\n",
    "\n",
    "For example, if I toss two coins, and $A$ means the first coin lands\n",
    "face up, and $B$ means the second coin lands face up, then\n",
    "\n",
    "${{\\mathrm{p}(A)}} =\n",
    "{{\\mathrm{p}(B)}} = 0.5$, and sure enough,\n",
    "${{\\mathrm{p}(A {~\\mathrm{and}~}B)}} = {{\\mathrm{p}(A)}}~{{\\mathrm{p}(B)}} = 0.25$.\n",
    "\n",
    "But this formula only works because in this case $A$ and $B$ are\n",
    "independent; that is, knowing the outcome of the first event does not\n",
    "change the probability of the second. Or, more formally,\n",
    "<span>$\\mathrm{p}(B|A)$</span> = <span>$\\mathrm{p}(B)$</span>.\n",
    "\n",
    "Here is a different example where the events are not independent.\n",
    "Suppose that $A$ means that it rains today and $B$ means that it rains\n",
    "tomorrow. If I know that it rained today, it is more likely that it will\n",
    "rain tomorrow, so\n",
    "\n",
    "${{\\mathrm{p}(B|A)}} > {{\\mathrm{p}(B)}}$.\n",
    "\n",
    "In general, the probability of a conjunction is\n",
    "$${{\\mathrm{p}(A {~\\mathrm{and}~}B)}} = {{\\mathrm{p}(A)}}~{{\\mathrm{p}(B|A)}}$$\n",
    "for any $A$ and $B$. So if the chance of rain on any given day is 0.5,\n",
    "the chance of rain on two consecutive days is not 0.25, but probably a\n",
    "bit higher."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The cookie problem\n",
    "\n",
    "We’ll get to Bayes’s theorem soon, but I want to motivate it with an\n",
    "example called the cookie problem.[^1] Suppose there are two bowls of\n",
    "cookies. Bowl 1 contains 30 vanilla cookies and 10 chocolate cookies.\n",
    "Bowl 2 contains 20 of each.\n",
    "\n",
    "Now suppose you choose one of the bowls at random and, without looking,\n",
    "select a cookie at random. The cookie is vanilla. What is the\n",
    "probability that it came from Bowl 1?\n",
    "\n",
    "This is a conditional probability; we want\n",
    "<span>$\\mathrm{p}({\\mbox{Bowl 1}} |\n",
    "  {\\mbox{vanilla}})$</span>, but it is not obvious how to compute it. If\n",
    "I asked a different question—the probability of a vanilla cookie given\n",
    "Bowl 1—it would be easy:\n",
    "\n",
    "$${{\\mathrm{p}({\\mbox{vanilla}} | {\\mbox{Bowl 1}})}} = 3/4$$\n",
    "\n",
    "Sadly, <span>$\\mathrm{p}(A|B)$</span> is <span>*not*</span> the same as\n",
    "<span>$\\mathrm{p}(B|A)$</span>, but there is a way to get from one to\n",
    "the other: Bayes’s theorem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayes’s theorem\n",
    "\n",
    "At this point we have everything we need to derive Bayes’s theorem.\n",
    "We’ll start with the observation that conjunction is commutative; that\n",
    "is\n",
    "\n",
    "$${{\\mathrm{p}(A {~\\mathrm{and}~}B)}} = {{\\mathrm{p}(B {~\\mathrm{and}~}A)}}$$\n",
    "for any events $A$ and $B$.\n",
    "\n",
    "Next, we write the probability of a conjunction:\n",
    "\n",
    "$${{\\mathrm{p}(A {~\\mathrm{and}~}B)}} = {{\\mathrm{p}(A)}}~{{\\mathrm{p}(B|A)}}$$\n",
    "\n",
    "Since we have not said anything about what $A$ and $B$ mean, they are\n",
    "interchangeable. Interchanging them yields\n",
    "\n",
    "$${{\\mathrm{p}(B {~\\mathrm{and}~}A)}} = {{\\mathrm{p}(B)}}~{{\\mathrm{p}(A|B)}}$$\n",
    "\n",
    "That’s all we need. Pulling those pieces together, we get\n",
    "\n",
    "$${{\\mathrm{p}(B)}}~{{\\mathrm{p}(A|B)}} = {{\\mathrm{p}(A)}}~{{\\mathrm{p}(B|A)}}$$\n",
    "\n",
    "Which means there are two ways to compute the conjunction. If you have\n",
    "<span>$\\mathrm{p}(A)$</span>, you multiply by the conditional\n",
    "probability <span>$\\mathrm{p}(B|A)$</span>. Or you can do it the other\n",
    "way around; if you know <span>$\\mathrm{p}(B)$</span>, you multiply by\n",
    "<span>$\\mathrm{p}(A|B)$</span>. Either way you should get the same\n",
    "thing.\n",
    "\n",
    "Finally we can divide through by <span>$\\mathrm{p}(B)$</span>:\n",
    "$${{\\mathrm{p}(A|B)}} = \\frac{{{\\mathrm{p}(A)}}~{{\\mathrm{p}(B|A)}}}{{{\\mathrm{p}(B)}}}$$\n",
    "And that’s Bayes’s theorem! It might not look like much, but it turns\n",
    "out to be surprisingly powerful.\n",
    "\n",
    "For example, we can use it to solve the cookie problem. I’ll write $B_1$\n",
    "for the hypothesis that the cookie came from Bowl 1 and $V$ for the\n",
    "vanilla cookie. Plugging in Bayes’s theorem we get\n",
    "\n",
    "$${{\\mathrm{p}(B_1|V)}} = \\frac{{{\\mathrm{p}(B_1)}}~{{\\mathrm{p}(V|B_1)}}}{{{\\mathrm{p}(V)}}}$$\n",
    "\n",
    "The term on the left is what we want: the probability of Bowl 1, given\n",
    "that we chose a vanilla cookie. The terms on the right are:\n",
    "\n",
    "-   ${{\\mathrm{p}(B_1)}}$: This is the probability that we\n",
    "    chose Bowl 1, unconditioned by what kind of cookie we got. Since the\n",
    "    problem says we chose a bowl at random, we can assume\n",
    "    ${{\\mathrm{p}(B_1)}} = 1/2$.\n",
    "\n",
    "-   ${{\\mathrm{p}(V|B_1)}}$: This is the probability of\n",
    "    getting a vanilla cookie from Bowl 1, which is 3/4.\n",
    "\n",
    "-   <span>$\\mathrm{p}(V)$</span>: This is the probability of drawing a\n",
    "    vanilla cookie from either bowl. Since we had an equal chance of\n",
    "    choosing either bowl and the bowls contain the same number of\n",
    "    cookies, we had the same chance of choosing any cookie. Between the\n",
    "    two bowls there are 50 vanilla and 30 chocolate cookies, so\n",
    "    <span>$\\mathrm{p}(V)$</span> = 5/8.\n",
    "\n",
    "Putting it together, we have\n",
    "\n",
    "$${{\\mathrm{p}(B_1|V)}} = \\frac{(1/2)~(3/4)}{5/8}$$ \n",
    "\n",
    "which\n",
    "reduces to 3/5. So the vanilla cookie is evidence in favor of the\n",
    "hypothesis that we chose Bowl 1, because vanilla cookies are more likely\n",
    "to come from Bowl 1.\n",
    "\n",
    "This example demonstrates one use of Bayes’s theorem: it provides a\n",
    "strategy to get from <span>$\\mathrm{p}(B|A)$</span> to\n",
    "<span>$\\mathrm{p}(A|B)$</span>. This strategy is useful in cases, like\n",
    "the cookie problem, where it is easier to compute the terms on the right\n",
    "side of Bayes’s theorem than the term on the left."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The diachronic interpretation\n",
    "\n",
    "There is another way to think of Bayes’s theorem: it gives us a way to\n",
    "update the probability of a hypothesis, $H$, in light of some body of\n",
    "data, $D$.\n",
    "\n",
    "This way of thinking about Bayes’s theorem is called the\n",
    "<span>**diachronic interpretation**</span>. “Diachronic” means that\n",
    "something is happening over time; in this case the probability of the\n",
    "hypotheses changes, over time, as we see new data.\n",
    "\n",
    "Rewriting Bayes’s theorem with $H$ and $D$ yields:\n",
    "$${{\\mathrm{p}(H|D)}} = \\frac{{{\\mathrm{p}(H)}}~{{\\mathrm{p}(D|H)}}}{{{\\mathrm{p}(D)}}}$$\n",
    "In this interpretation, each term has a name:\n",
    "\n",
    "-   <span>$\\mathrm{p}(H)$</span> is the probability of the hypothesis\n",
    "    before we see the data, called the prior probability, or just\n",
    "    <span>**prior**</span>.\n",
    "\n",
    "-   <span>$\\mathrm{p}(H|D)$</span> is what we want to compute, the\n",
    "    probability of the hypothesis after we see the data, called the\n",
    "    <span>**posterior**</span>.\n",
    "\n",
    "-   <span>$\\mathrm{p}(D|H)$</span> is the probability of the data under\n",
    "    the hypothesis, called the <span>**likelihood**</span>.\n",
    "\n",
    "-   <span>$\\mathrm{p}(D)$</span> is the probability of the data under\n",
    "    any hypothesis, called the <span>**normalizing constant**</span>.\n",
    "\n",
    "Sometimes we can compute the prior based on background information. For\n",
    "example, the cookie problem specifies that we choose a bowl at random\n",
    "with equal probability.\n",
    "\n",
    "In other cases the prior is subjective; that is, reasonable people might\n",
    "disagree, either because they use different background information or\n",
    "because they interpret the same information differently.\n",
    "\n",
    "The likelihood is usually the easiest part to compute. In the cookie\n",
    "problem, if we know which bowl the cookie came from, we find the\n",
    "probability of a vanilla cookie by counting.\n",
    "\n",
    "The normalizing constant can be tricky. It is supposed to be the\n",
    "probability of seeing the data under any hypothesis at all, but in the\n",
    "most general case it is hard to nail down what that means.\n",
    "\n",
    "Most often we simplify things by specifying a set of hypotheses that are\n",
    "\n",
    "Mutually exclusive:\n",
    ":   At most one hypothesis in the set can be true, and\n",
    "\n",
    "Collectively exhaustive:\n",
    ":   There are no other possibilities; at least one of the hypotheses has\n",
    "    to be true.\n",
    "\n",
    "I use the word <span>**suite**</span> for a set of hypotheses that has\n",
    "these properties.\n",
    "\n",
    "In the cookie problem, there are only two hypotheses—the cookie came\n",
    "from Bowl 1 or Bowl 2—and they are mutually exclusive and collectively\n",
    "exhaustive.\n",
    "\n",
    "In that case we can compute <span>$\\mathrm{p}(D)$</span> using the law\n",
    "of total probability, which says that if there are two exclusive ways\n",
    "that something might happen, you can add up the probabilities like this:\n",
    "\n",
    "$${{\\mathrm{p}(D)}} = {{\\mathrm{p}(B_1)}}~{{\\mathrm{p}(D|B_1)}} + {{\\mathrm{p}(B_2)}}~{{\\mathrm{p}(D|B_2)}}$$\n",
    "\n",
    "Plugging in the values from the cookie problem, we have\n",
    "\n",
    "$${{\\mathrm{p}(D)}} = (1/2)~(3/4) + (1/2)~(1/2) = 5/8$$ \n",
    "\n",
    "which is what we computed earlier by mentally combining the two bowls."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The <span>M&M</span> problem\n",
    "\n",
    "<span>M&M</span>’s are small candy-coated chocolates that come in a\n",
    "variety of colors. Mars, Inc., which makes <span>M&M</span>’s, changes\n",
    "the mixture of colors from time to time.\n",
    "\n",
    "In 1995, they introduced blue <span>M&M</span>’s. Before then, the color\n",
    "mix in a bag of plain <span>M&M</span>’s was 30% Brown, 20% Yellow, 20%\n",
    "Red, 10% Green, 10% Orange, 10% Tan. Afterward it was 24% Blue , 20%\n",
    "Green, 16% Orange, 14% Yellow, 13% Red, 13% Brown.\n",
    "\n",
    "Suppose a friend of mine has two bags of <span>M&M</span>’s, and he\n",
    "tells me that one is from 1994 and one from 1996. He won’t tell me which\n",
    "is which, but he gives me one <span>M&M</span> from each bag. One is\n",
    "yellow and one is green. What is the probability that the yellow one\n",
    "came from the 1994 bag?\n",
    "\n",
    "This problem is similar to the cookie problem, with the twist that I\n",
    "draw one sample from each bowl/bag. This problem also gives me a chance\n",
    "to demonstrate the table method, which is useful for solving problems\n",
    "like this on paper. In the next chapter we will solve them\n",
    "computationally.\n",
    "\n",
    "The first step is to enumerate the hypotheses. The bag the yellow\n",
    "<span>M&M</span> came from I’ll call Bag 1; I’ll call the other Bag 2.\n",
    "So the hypotheses are:\n",
    "\n",
    "-   A: Bag 1 is from 1994, which implies that Bag 2 is from 1996.\n",
    "\n",
    "-   B: Bag 1 is from 1996 and Bag 2 from 1994.\n",
    "\n",
    "Now we construct a table with a row for each hypothesis and a column for\n",
    "each term in Bayes’s theorem:\n",
    "\n",
    "|  |Prior  $\\mathrm{p}(H)$ | Likelihood $\\mathrm{p}(D\\vert H)$ | $\\mathrm{p}(H) \\mathrm{p}(D\\vert H)$ |  Posterior $\\mathrm{p}(H\\vert D)$|\n",
    "| --- | ---------------- | --------------------------- | ----------- | ------ |\n",
    "| A | 1/2 | (20)(20) | 200 | 20/27|\n",
    "| B | 1/2 | (14)(10) | 70  |   7/27|\n",
    "\n",
    "\n",
    "The first column has the priors. Based on the statement of the problem,\n",
    "it is reasonable to choose\n",
    "${{\\mathrm{p}(A)}} = {{\\mathrm{p}(B)}} = 1/2$.\n",
    "\n",
    "The second column has the likelihoods, which follow from the information\n",
    "in the problem. For example, if $A$ is true, the yellow\n",
    "<span>M&M</span> came from the 1994 bag with probability 20%, and the\n",
    "green came from the 1996 bag with probability 20%. If $B$ is true, the\n",
    "yellow <span>M&M</span> came from the 1996 bag with probability 14%, and\n",
    "the green came from the 1994 bag with probability 10%. Because the\n",
    "selections are independent, we get the conjoint probability by\n",
    "multiplying.\n",
    "\n",
    "The third column is just the product of the previous two. The sum of\n",
    "this column, 270, is the normalizing constant. To get the last column,\n",
    "which contains the posteriors, we divide the third column by the\n",
    "normalizing constant.\n",
    "\n",
    "That’s it. Simple, right?\n",
    "\n",
    "Well, you might be bothered by one detail. I write\n",
    "<span>$\\mathrm{p}(D|H)$</span> in terms of percentages, not\n",
    "probabilities, which means it is off by a factor of 10,000. But that\n",
    "cancels out when we divide through by the normalizing constant, so it\n",
    "doesn’t affect the result.\n",
    "\n",
    "When the set of hypotheses is mutually exclusive and collectively\n",
    "exhaustive, you can multiply the likelihoods by any factor, if it is\n",
    "convenient, as long as you apply the same factor to the entire column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Monty Hall problem\n",
    "\n",
    "The Monty Hall problem might be the most contentious question in the\n",
    "history of probability. The scenario is simple, but the correct answer\n",
    "is so counterintuitive that many people just can’t accept it, and many\n",
    "smart people have embarrassed themselves not just by getting it wrong\n",
    "but by arguing the wrong side, aggressively, in public.\n",
    "\n",
    "Monty Hall was the original host of the game show <span>*Let’s Make a\n",
    "Deal*</span>. The Monty Hall problem is based on one of the regular\n",
    "games on the show. If you are on the show, here’s what happens:\n",
    "\n",
    "-   Monty shows you three closed doors and tells you that there is a\n",
    "    prize behind each door: one prize is a car, the other two are less\n",
    "    valuable prizes like peanut butter and fake finger nails. The prizes\n",
    "    are arranged at random.\n",
    "\n",
    "-   The object of the game is to guess which door has the car. If you\n",
    "    guess right, you get to keep the car.\n",
    "\n",
    "-   You pick a door, which we will call Door A. We’ll call the other\n",
    "    doors B and C.\n",
    "\n",
    "-   Before opening the door you chose, Monty increases the suspense by\n",
    "    opening either Door B or C, whichever does not have the car. (If the\n",
    "    car is actually behind Door A, Monty can safely open B or C, so he\n",
    "    chooses one at random.)\n",
    "\n",
    "-   Then Monty offers you the option to stick with your original choice\n",
    "    or switch to the one remaining unopened door.\n",
    "\n",
    "The question is, should you “stick” or “switch” or does it make no\n",
    "difference?\n",
    "\n",
    "Most people have the strong intuition that it makes no difference. There\n",
    "are two doors left, they reason, so the chance that the car is behind\n",
    "Door A is 50%.\n",
    "\n",
    "But that is wrong. In fact, the chance of winning if you stick with Door\n",
    "A is only 1/3; if you switch, your chances are 2/3.\n",
    "\n",
    "By applying Bayes’s theorem, we can break this problem into simple\n",
    "pieces, and maybe convince ourselves that the correct answer is, in\n",
    "fact, correct.\n",
    "\n",
    "To start, we should make a careful statement of the data. In this case\n",
    "$D$ consists of two parts: Monty chooses Door B <span>*and*</span> there\n",
    "is no car there.\n",
    "\n",
    "Next we define three hypotheses: $A$, $B$, and $C$ represent the\n",
    "hypothesis that the car is behind Door A, Door B, or Door C. Again,\n",
    "let’s apply the table method:\n",
    "\n",
    "\n",
    "|  |Prior  $\\mathrm{p}(H)$ | Likelihood $\\mathrm{p}(D\\vert H)$ | $\\mathrm{p}(H) \\mathrm{p}(D\\vert H)$ |  Posterior $\\mathrm{p}(H\\vert D)$|\n",
    "| --- | ---------------- | --------------------------- | ----------- | ------ |\n",
    "|   A | 1/3  | 1/2   | 1/6 |1/3|\n",
    "|   B | 1/3 | 0 | 0  | 0 |\n",
    "|   C | 1/3 | 1  | 1/3  | 2/3 |\n",
    "\n",
    "Filling in the priors is easy because we are told that the prizes are\n",
    "arranged at random, which suggests that the car is equally likely to be\n",
    "behind any door.\n",
    "\n",
    "Figuring out the likelihoods takes some thought, but with reasonable\n",
    "care we can be confident that we have it right:\n",
    "\n",
    "-   If the car is actually behind A, Monty could safely open Doors B or\n",
    "    C. So the probability that he chooses B is 1/2. And since the car is\n",
    "    actually behind A, the probability that the car is not behind B is\n",
    "    1.\n",
    "\n",
    "-   If the car is actually behind B, Monty has to open door C, so the\n",
    "    probability that he opens door B is 0.\n",
    "\n",
    "-   Finally, if the car is behind Door C, Monty opens B with probability\n",
    "    1 and finds no car there with probability 1.\n",
    "\n",
    "Now the hard part is over; the rest is just arithmetic. The sum of the\n",
    "third column is 1/2. Dividing through yields\n",
    "${{\\mathrm{p}(A|D)}} = 1/3$ and\n",
    "${{\\mathrm{p}(C|D)}} = 2/3$. So you are better off switching.\n",
    "\n",
    "There are many variations of the Monty Hall problem. One of the\n",
    "strengths of the Bayesian approach is that it generalizes to handle\n",
    "these variations.\n",
    "\n",
    "For example, suppose that Monty always chooses B if he can, and only\n",
    "chooses C if he has to (because the car is behind B). In that case the\n",
    "revised table is:\n",
    "\n",
    "|  |Prior  $\\mathrm{p}(H)$ | Likelihood $\\mathrm{p}(D\\vert H)$ | $\\mathrm{p}(H) \\mathrm{p}(D\\vert H)$ |  Posterior $\\mathrm{p}(H\\vert D)$|\n",
    "| --- | ---------------- | --------------------------- | ----------- | ------ |\n",
    "|   A | 1/3| 1 |1/3  | 1/2\n",
    "|   B |1/3 |  0 | 0    |  0 |\n",
    "|   C |1/3 | 1 | 1/3 | 1/2|\n",
    "\n",
    "The only change is <span>$\\mathrm{p}(D|A)$</span>. If the car is behind\n",
    "$A$, Monty can choose to open B or C. But in this variation he always\n",
    "chooses B, so ${{\\mathrm{p}(D|A)}} = 1$.\n",
    "\n",
    "As a result, the likelihoods are the same for $A$ and $C$, and the\n",
    "posteriors are the same:\n",
    "${{\\mathrm{p}(A|D)}} = {{\\mathrm{p}(C|D)}} = 1/2$.\n",
    "In this case, the fact that Monty chose B reveals no information about\n",
    "the location of the car, so it doesn’t matter whether the contestant\n",
    "sticks or switches.\n",
    "\n",
    "On the other hand, if he had opened $C$, we would know\n",
    "${{\\mathrm{p}(B|D)}} = 1$.\n",
    "\n",
    "I included the Monty Hall problem in this chapter because I think it is\n",
    "fun, and because Bayes’s theorem makes the complexity of the problem a\n",
    "little more manageable. But it is not a typical use of Bayes’s theorem,\n",
    "so if you found it confusing, don’t worry!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "\n",
    "For many problems involving conditional probability, Bayes’s theorem\n",
    "provides a divide-and-conquer strategy. If\n",
    "<span>$\\mathrm{p}(A|B)$</span> is hard to compute, or hard to measure\n",
    "experimentally, check whether it might be easier to compute the other\n",
    "terms in Bayes’s theorem, <span>$\\mathrm{p}(B|A)$</span>,\n",
    "<span>$\\mathrm{p}(A)$</span> and <span>$\\mathrm{p}(B)$</span>.\n",
    "\n",
    "If the Monty Hall problem is your idea of fun, I have collected a number\n",
    "of similar problems in an article called “All your Bayes are belong to\n",
    "us,” which you can read at\n",
    "<http://allendowney.blogspot.com/2011/10/all-your-bayes-are-belong-to-us.html>."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
